Repository navigation
fix(connectors): validate and repair the knowledge-base connector fleet - #6757
Merged
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five review passes over all 61 knowledge-base connectors, validated against each provider's live documentation or machine-readable spec. Passes 3 through 5 reviewed the previous pass's edits, which is where most of what follows was caught — including regressions the audit itself introduced.
The dominant bug class
shouldReconcileDeletionshard-deletes any stored document absent from a listing that did not setsyncContext.listingCapped. So any path that turns a failure into an empty or short listing is a silent mass-delete. Most fixes here are instances of that one shape:startand readtotalSize; neither exists on/content/search.hasMorecould never be true, so it only ever returned the first page, and that truncated listing fed deletion reconciliation on every sync.labels.listwas swallowed, producing a query matching nothing. An empty page arriving with anextPageTokenalso ended the listing early, producing a short-but-complete-looking listing that reconciliation acted on. Separately, the labels selector stores label ids whilevalidateConfigcompared against label names, so any user label was rejected at save time and could never be configured at all (system labels, where id equals name, worked).listingCappedwas set nowhere, andmaxEventsdefaults to 500. Any calendar with more than 500 events in the configured window listed only the first ~500 and reconciliation hard-deleted the rest — on the default configuration, with nothing for the user to misconfigure.listingCappednowhere, so every stored document beyond the cap was hard-deleted on every run. wordpress additionally paged by pure offset, which re-numbers mid-run when a post is published, skipping posts.formatDateused local-time getters against an API documented in UTC, so on any non-UTC host the window boundary shifted by a day and that day's recordings could fall outside every window and never be listed. Separately, both bounds are inclusive but the span was computed asto - WINDOW_DAYS, producing 31-day windows that overlapped by a day at each seam — seam-day recordings were listed twice and consumed the budget twice.LIMITbounded the entire result set with nonextRecordsUrl, so any org larger than that limit had the remainder hard-deleted every sync.listingCappedwas set nowhere, so a repo truncated by the Trees API's own 100,000-entry / 7 MB limit, or by amaxFilescap, produced a short listing and reconciliation hard-deleted the stored documents outside it.documents.length < stubs.length, mutation-verified.maxMessagesproducedNaN, the loop never ran, and the empty listing hard-deleted the stored document.Other production faults repaired
API_VERSIONwas'v62.0'substituted into a{version}placeholder that already carries thev, yielding/services/data/vv62.0/. Every REST call 404'd.incompleteSearchwas read in code but never named in the Drivefieldsmask, so the partial-corpus guard was dead code.listingCappedwas set nowhere, so any source withmaxIssuesbelow its real issue count listed only the top N byupdated DESCand hard-deleted everything that fell out of that window, every sync. (Correcting an earlier draft of this description: the/search/jqlmigration is not in this diff — staging already used that endpoint. OnlyPAGE_SIZEchanged.)Cap-at-exhaustion, and the deletion wave it implies
The inverse error: setting
listingCappedwhen the cap lands exactly on source exhaustion permanently blocks reconciliation, so documents deleted upstream are stranded in the knowledge base forever. Fixed across gmail, ashby, azure-devops, grain, granola, greenhouse, incidentio and others.This has a visible consequence on the first sync. Those connectors have been over-flagging, so deletion reconciliation has been suppressed on them — possibly since setup. Once the gate is correct, the documents that were stranded become eligible for removal and will reconcile away. That is the fix working as intended, and it is two-phase (tombstone, then delete only on a second consecutive absence, behind the suspect-listing guard), but it will read as a deletion wave.
Two connectors were worse than over-flagging: hubspot and intercom never set
listingCappedat all, so every capped sync hard-deleted every stored document beyond the cap, on every run. Both now set it.Pagination
"Last-page precision" (shrinking page size to the remaining budget) is only valid when the cursor is position-independent. Over an opaque cursor token no provider documents that a cursor survives a changed page size, and the cap was already enforced server-side — so it bought nothing and risked a mid-sync iterator error. Reverted on airtable, asana, ashby and confluence.
To be precise about scope, since an earlier draft of this description overstated it: no taper was ever committed for greenhouse or intercom. Both were audited as page-number APIs (Harvest documents
pageas "the n-th chunk ofper_pageobjects", and Intercom's/articlesis the same shape), and both already held their page size constant on staging. The only change there is a comment recording why it must stay constant — there was nothing to revert.Regressions this audit introduced and fixed within it
Net zero versus staging. Listed because the record should be honest about which findings were pre-existing and which were our own — a later pass reviewed each earlier pass's edits, which is the only reason these did not ship.
.catch(() => null)on the JSON parse plusdata?.data ?? {}, turning an unparseable 2xx into a confident empty listing. Staging returneddata.databare and threw loudly, so this was never a production fault.matchesExtensionin a way that dropped dotfiles, and separately added autf-8blob branch justified by a misattributed quote. Both reverted; net versus staging is TSDoc only.:partialto a hash that then never converged.404 → breakin the newfetchParticipants; staging had no such function.startedenum member; staging omitted the parameter entirely.secure-fetch— retry headers attached enumerably, so the logger's own-enumerable copy wrote upstreamSet-Cookievalues into sync logs. Staging attached no headers at all.Security
SecureFetchHeadersdeclaresprivate setCookies: string[], but TypeScriptprivateis compile-time only — at runtime it is an own enumerable property.@sim/loggercopies own enumerable properties into the formatted line and the retry loop logs{ error }on every failed attempt, so newly-attached response headers wrote upstreamSet-Cookievalues into connector-sync logs. Now attached non-enumerably. Scope is the dev/colorize branch only — production reduces errors to.message.SFTP now requires a host-key fingerprint on all sources, failing closed before the credential is sent (verified by mutation: 7 tests go red). DocuSign's OAuth host is allowlisted to demo/production so a misconfigured value cannot redirect the flow and its bearer token to another origin.
Fabricated doc citations
Eight TSDoc comments quoted provider sentences that do not exist (github ×2, jira, jsm, linear, google-meet, dropbox, youtube) — two of them introduced by this audit. One inverted the usual failure mode: google-meet's code was correct and the comment invented a sentence to justify it, which nearly got a correct fallback reverted. Worth noting that two of three "unverifiable" flags carried into the final pass turned out to be fully documented — re-checking beat reverting on suspicion.
Removed
Evernote, entirely — its auth flow was non-functional, so this is removal rather than repair. Verified complete: the only surviving references are prose in agent-tooling markdown; both icon registries are de-registered. No knowledge-base source, no deployed workflow, and no executed workflow references it.
Customer impact
One-time re-index, deliberate.
externalIdis unchanged in all three, so these are in-place updates — no deletes, no re-creates, no re-auth:Other behaviour changes worth watching on first sync:
getDocumentnow throws instead of returning null, so persistent content-read failures become visible failed rows. Expect a one-time rise in reported failures; this is the intended invariant, not a defect.MONDAY_API_VERSIONmoves four call sites (both selector routes, and webhook create/delete) from an inline2024-10pin to2026-04. To be precise:2024-10is in Maintenance, not deprecated, so it was still being served as declared — this is not a deprecation fix. What was genuinely wrong is the divergence: the selector routes listed boards at one schema version while the tools operating on those boards ran at another, inside a single user-visible flow. The consolidation is right, but it jumps roughly five version bumps at those four sites, and monday fails silently here (an unrecognized version resolves to Current rather than erroring). Worth a reviewer's eye rather than being read as a pure refactor.rtffrom the default extension list and excludes keys containing./..path segments. Both are intentional scope filters, so they correctly do not setlistingCappedand the affected documents reconcile away.Operational notes for the first post-deploy sync
Two things will look alarming in logs and are not incidents.
notion's re-index will not finish in one task window. The corpus must re-hydrate against a documented ~3 req/s average while the engine hydrates 5 documents concurrently (
SYNC_BATCH_SIZE = 5) against a 30-minute budget (maxDuration: 1800). Expect the run to be killed mid-sync with heavy 429 backoff first. It is not destructive and needs no intervention: batches commit as they go, a page that exhausts its retries rejects and is therefore excluded from deletion reconciliation, and a re-hydrated page classifiesunchangednext run. It should converge over several scheduled syncs.jira's deletion reconciliation will run for the first time on mid-sized projects.
PAGE_SIZE50 → 100 raises the effective listing ceiling from 25,000 to 50,000 (MAX_PAGES = 500). A project in the 25,001–50,000 band previously always hit the ceiling, which setlistingTruncated— an unconditional, non-overridable block on reconciliation — so stale rows have accumulated since setup. Those projects now list completely and will tombstone that backlog on the first full sync. This is the correct outcome and it is two-phase behind the suspect-listing guard, but note the honest framing: raising the page size enables deletion for that band rather than fixing a deletion bug. A project larger than 50,000 truncates again and still fails safe.Final validation sweep
A last read-only
/validate-connectorpass ran over all 59 changed connectors plus the shared files, verifying every quoted claim against provider specs. It found six real defects, all now fixed:getDocumentreturnednullon an API-shape fault. Both connectors setcontentDeferred, so these paths were live: a fulfillednullrecords no failure and no log, and the document silently vanishes.LookupErrorunion, andrestricted_content/lockedboth mean the file still exists. Onlynot_foundis absence now.fetchFormValuesswallowed every non-OK status, baking a permanently incomplete document (the hash is metadata-only, so the next sync sees it asunchangedforever).allsentresponse_type=started,partial,completed, but Typeform documents onlypartialandcompleted. An unknown enum member risks a 400 that fails the whole sync, and staging omitted the parameter entirely — so this shipped as a regression. Now sends the widest documented set, which is strictly wider than staging's effective behaviour.utf-8blob branch was added on a misattributed quote. That sentence describes theencodingrequest parameter of Create a blob; the GET response is documented as always base64. Branch removed, and two comments corrected that were hiding a real drop (>1 MB files undervnd.github+json403 rather than returningencoding: none).The sweep also confirmed the
htmlToPlainTextblast radius independently: exactly microsoft-teams and mintlify both decode HTML and hash the result, verified by intersecting thehtmlToPlainTextimporters against thecomputeContentHashcallers.The corollary matters more than the re-index does: for the other ~22 connectors that call
htmlToPlainText, the hash is metadata-keyed, so the entity-decode fix is forward-only — already-indexed documents keep their literal’text until the upstream record's own timestamp changes. The same is true of the Jira ADF extraction improvements, since bothjiraandjsmhash on metadata. Correcting the existing corpus would need a deliberate rehydrate, not this deploy.hubspot additionally moved to a shared anchored HTML-detection helper. The loose pattern matched angle-bracketed prose (
Reply from John <john@acme.com>), andhtmlToPlainTextdeletes the span and collapses line structure — so a false positive loses data rather than passing it through. Worth fixing now rather than later precisely because thehubspot:v2:bump rewrites every affected document once.Adversarial re-validation
A final pass re-ran
/validate-connectorper connector against the current branch state and put every finding in front of an independent agent whose job was to refute it. Zero blockers survived. One major and a set of minors did; two were fixed here.buildWiqlorders byChangedDate DESCwhile work-item ids are assigned in creation order, so the highest-id match is almost always inside the returned window. The probe therefore came back empty for genuinely truncated projects, left the listing unflagged, and let deletion reconciliation remove every indexed item outside it — on the default, non-incremental sync path. Now flagged unconditionally. The trade is that a project sitting exactly at the ceiling stops reconciling deletions until a full resync; the alternative was silent data loss on every scheduled sync. Proper pagination (order by[System.Id] ASC, loop onidAfter) is left as a seam with the parameter and a TSDoc pointing at it.fetchFormValuesthrew on a non-404 status and then caught its own throw, returning[]. An earlier fix in this PR was a no-op; the catch now rethrows.Honest caveat on that pass: it hit a session limit partway through, so roughly the second half of the fleet alphabetically had its refutation stage killed. Those connectors' findings were dropped rather than adjudicated, and the pass reports them as clean — they are unadjudicated, not verified. The findings themselves are recorded and none was a blocker in the first-half sample.
The one item I'd want a decision on before merge
sentry— the endpoint migration narrows scope and will delete issues. Moving from the deprecated/projects/{org}/{project}/issues/to/organizations/{org}/issues/?project=is correct on its own terms, and it fixes a genuine bug alongside it (staging sorted on Sentry's defaultdate, a mutable key, so under cursor paging an issue whoselastSeenadvanced mid-sync jumped the cursor, was skipped, and was hard-deleted as absent).But the organization endpoint always resolves a date range, and with no
statsPeriod/start/endit defaults to roughly 90 days. Issues last seen before that window are absent from the listing and reconcile away. That is a scope narrowing this PR introduces, not one it inherits — the in-code comment currently rationalizes it as the pre-existing "aged out of the query window" semantic, which understates it.I did not set an explicit window because I could not verify Sentry's maximum accepted value, and a wrong one 400s the endpoint. There are no live Sentry sources, so nothing is at risk today. Options are: accept it as intended scope, pin an explicit widest-supported window, or set
listingCappedwhen the window is the limiting factor. Worth an owner's call rather than my guess.Known gaps
All 61 connectors have been through the final pass, verified against the four invariants — cap gating,
getDocumentabsence semantics,contentHashparity, and pagination shape.isScopeSatisfiedBy(lib/oauth/utils.ts) was the one shared-OAuth item flagged as needing scrutiny — it treats a granted non-readonly scope as satisfying its.readonlyvariant by string suffix, which would be unsound if any provider had afoo.readonlyscope wherefoois not a superset. Checked and clear. Exactly three.readonlyrequiredScopes exist fleet-wide, all Google, and all three appear verbatim in their provider scope lists, so exact match already satisfies them. The suffix rule fires for only one:ediscovery(bare) is grantable and is a genuine documented superset ofediscovery.readonly. The other two cannot trigger it — baremeetings.spaceandforms.responsesare not real scopes and appear in no provider list. No false-positive path exists.Issues found and deliberately not fixed here, each with a reason:
outlookroutes a correctness-critical value betweenlistDocumentsandgetDocumentthrough an untypedsyncContextkey thatconnectors/types.tsdocuments as a cache. Correct today — the engine has exactly onegetDocumentcaller — but the fallback fails silently and permanently if that ever changes.listingCappedand skips the tombstone grace period, hard-deleting hidden documents in one pass. Correct semantics; deserves a UI warning.gongmaps a 404 to an empty page unconditionally, which can hard-delete a source over two consecutive syncs. The fix requires matching a documented error string that could not be verified — Gong's API reference is not publicly fetchable — and guessing at it would be worse than the known behaviour.rootlyswallows timeline-fetch failures into an empty array, baking a permanently timeline-less document (the hash isupdated_at-keyed, so it never self-repairs). Pagerduty solved this class with an explicit failure flag; rootly is the remaining instance.gitlabandnotionflaglistingCappedwhen a count merely reaches the cap without confirming more remained. Errs safe — over-suppression, never over-deletion — and pre-existing.servicenowsorts by a mutable key under offset pagination, so a record updated mid-sync can slide past the window and read as deleted. Sentry's equivalent was fixed here by switching to an immutable sort key.youtubebuildscontentHashfrom different fields on the stub and hydrated paths. Tracing the engine shows it settles todocsUnchangedrather than re-indexing, so the cost is a wasted API call per video per sync — quota burn, not churn.Verification
29 audits (
check:audits), 24/24 type-check, 1,332 tests, CodeQL clean, biome clean.